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ABSTRACT 

This paper describes the Multiple Assessment Programs 
and Services (MAPS) Placement Research Service. Students entering 
nonselective colleges show a wide range of ability. The College Board 
offers these colleges a number of tests of reading, writing, and 
mathematics skills for use in placing students appropriately. The 
NAPS Placement Research Service is a new service available to 
colleges that use these tests. The service analyzes test score and 
criterion data sipplied by the college and produces a report based on 
the data. The cou.lege can use the information to establish decision 
rules for placing students into courses or to counsel individual 
students in their selection of courses. The MAP Placement Research 
Service report consists entirely of tables, is organized by groups of 
students, and includes the score distributions of all predictors and 
criteria and the intercorrelations of all predictors and criteria in 
each group of students. The user can request a series of detailed 
analyses of the relationship between particular predictors and 
criteria. For each analysis, the user specifies the group of 
students, the predictor or predictors, and the criterion measure. 
Tables, smoothing procedures, and sample sizes are discussed. A 
sample MAPS Plscement Research Service Report is appended. (PN) 
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A paper to be presented at the Annual Meeting of the National Council on 
Measurement in Education, San Francisco, April, 1986. 

Most people who think about educational testing probably think of the 
College Board as the organization responsible for the admissions tests used 
at selective colleges— the Scholastic Aptitude Test and the College Board 
Achievement Tests. However, many colleges and community colleges are not at 
all selective. Th€ students entering these colleges show a very wide range 
of ability in reading, writing, and mathematical skills, and they need to be 
placed in courses appropriate to their ability. The College Board offers 
these colleges a number of tests of reading, writing, and math skills, for 
use in placing students appropriately. These tests are part of "MAPS", 
which stands for "Multiple Assessment Programs and Services." The MAPS 
Placement Research Service is a new service that the College Board is making 
available to colleges that use these tests. The service consists of 
analyzing test score and criterion data supplied by the college and 
producing a report based on the data. The college can use the information 
in the report to establish decision rules for placing students into courses 
or to counsel individual students in their selection of courses. 

A MAPS Placement Research Service report consists entirely of tables. 
We considered supplementing the tables with computer-produced graphs, but we 
decided that the information the Placement Research Service produces could 
be presented quite clearly without any graphics. The additional benefit did 
not justify the additional costs. The first time each type of table appears 
in the report, it is accompanied by a brief explanation of the statistics it 
contains, on the page Just before the table. 
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A MAPS Placement Research Service report is organized by groups of 
students— first, the total group of all students, followed by whatever 
subgroups the user has specified. The system will produce separate analyses 
for up to six subgroups. The report always includes the score distributions 
of all predictors and criteria and the intercorrelations of all predictors 
and criteria in each group of students. Then, in addition, the user can 
request a series of detailed analyses of the relationships between 
particular predictors and criteria. For each analysis, the user specifies 
the group of students, the predictor or predictors, and the criterion 
measure . 

For each single-predictor analysis ti.e user requests, the system prints 
out four types of tables. The first is a two-way table of observed data— a 
contingency tabl« showing how many students had predictor scores in a 
particular interval and criterion scores in a particular interval. The user 
can eit'ier specify the score intervals or let the system do it for him. 

The second type of table is a 2x2 table of observed data. This is 
simply a reduced version of the full two-way table, and it is optional. In 
order to get it, the user has to specify a cutoff score on the criterion 
measure. The user may also specify a cutoff on the predictor, or else let 
the computer make the choice. fThe computer will choose the predictor 
cutoff score that makes the marginal frequencies on the predictor as close 
as possible to what they are on the criterion.) 

The third type of table for each single-predictor analysis is an 
expectancy table. This is a table of estimated conditional probabilities, 
for example, the probability that a student with a score of 41 to 45 on the 
writing test will obtain a grade of C or better in the regular first-year 
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English composition course. This table is probably the most useful and the 
most interesting of the tables the system produces. Useful, because It 
offers a kind of information that the test users can easily apply and under- 
stand. Interesting, because the statistical procedure for estimating the 
probabilities Is new. (I'll say more about the expectancy tables later.) 

The fourth type of table for each slngle-predlctor analysis Is a 
prediction equation, computed by ordinary leasts-squares regression. In 
addition to the prediction equation, the table also shows the sample size, 
the correlation, and the residual standard deviation. 

The tables that the system produces for a multiple-predictor analysis 
are someirhat different. There is no expectancy table. We considered the 
options for producing an expectancy tabxe based on multiple predictors. 
Conditioning on all predictors would be far too complex for the users. We 
could condition on the best two predictors. Or we could condition on the 
composite predictor that predicts best in the sample of scudents. None of 
these options seemed very attractive, and we finally decided against all of 
them. We don't produce an expectancy table for a multiple-predictor 
analysis. The user can get an expectancy table for a composite of two or 
three equally-weighted predictors by requesting a single-predictor analysis 
with the sun of these two or three variables as the single predictor. 

The tables for a multiple-predictor analysis begin with a table of step- 
wise regression equations, starting with the best single predictor and 
adding one predictor at a time. If any of the specified predictors make too 
small a contribution, they will not be included. Instead, the computer will 
print out a message that says "The following predictors were not included in 
the prediction equation because their additional predictive power in this 
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group of students was too small." However, the table does Include a 
prediction equation that includes all the specified predictors, weighted 
equally. Of course. If the predictors are on different scales, their 
effective weights will not be equal. But Inmost cases, this equal-velghted 
composite will produce almost as high a correlation as the empirically 
derived weights. 

Each prediction equation Is accompanied oy two correlations — the 
multiple R In the sample and an estimate of what the correlation using those 
same prediction weights would be In the population. 

The second type of table produced for each multiple-predictor analysis 
Is a two-way table of predicted scores vs. actual scores on the criterion 
measure, for the students In the sample. One feature of the data that Is 
quite apparent from this table Is how much less variation there Is In the 
predicted scores than In the actual scores. 

The third type of table for each multiple-predictor analysis Is a 2x2 
table of predicted scores vs. actual scores on the criterion measure, for 
the students In the sample. Again, this table Just presents a regrouping of 
the Information In the previous table. The 2x2 table showt9 the number of 
"hits" and "misses". In the sample used to derive the prediction equation. 
To get this table, the user has to specify a cutoff score on the criterion 
measure. This cutoff score is applied to both the predicted scores and the 
actual scores. 

The feature of the MAPS Placement Research Service that is most 
interesting from a statistical point of view is the way the expectancy 
tables are produced. They are produced by smoothing the observed two-way 
tables f using an iterative log-linear maximum-likelihood procedure developed 
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at ETS by Paul Rosenbaum and Dorothy Thayer. The smoothed table that this 
procedure produces has cell frequencies that are generally not whole 
numbers. Therefore, a cell frequency In the smoothed table Is a fractional 
number of stif.dents. This fractional cell frequency can be interpreted as an 
estimate of the expected number of students In that cell, averaging over 
repeated samples. That Is, ve consider the students In the observed two-way 
table as a sample from a population of students, and we try to estimate what 
the two-way table would look like, averaged over repeated samples from this 
population. Then, by summing over values of the criterion measure above a 
certain level, we get an estimate of the probability that a student from 
this population, with a predictor score In a certain Interval, would achieve 
at least a certain score on the criterion measure. 

The smoothing procedure offers a choice of models— stronger models for 
sparse data, weaker models for denser data— but none of the models Is so 
strong as to specify the shape of the distribution. We no longer have to 
pretend that the world Is blvarlate normal. Each model produces the 
•^smoothest" solution that preserves certain aspects of the original 
unsmoothed two-way table. The unsmoothed two-way table Is the same table 
the system prints out as the "two-way table of observed data"; It contains 
the same data, grouped Into the same Intervals on each variable. The 
strongest model preserves only the means, the variances, and the correlation 
between the predictor and criterion. That Is, the smoothing Is constrained 
so that the smoothed two-way table will have the same means, variances, and 
correlation between variables as the unsmoothed table, but everything else 
gets "smoothed out". We would generally not use this strong a model, 
however. The model we typically use In the Placement Research Service 
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preserves the meant the variance t and the skewness of each variable, and 
also the correlation between the tvo variables. There are weaker models 
that preserve the marginal frequencies of the unsmoothed table, but we would 
not use these models unless the sample were so large that the marginal 
distributions In the unsmoothed table were already fairly smooth. 

The handout contains an example of an observed two-*way table and the 
expectancy table produced from It. You can see that even where there if^ a 
zero cell frequency In the observed table, the smoothing procedure has 
estimated a positive probability of finding a student In that cell. (That 
Is, the corresponding cell In the expectancy table does not show the same 
probability as the cell to Its left.) 

In the MAPS Placement Research Service we deal with the problem of small 
sample size In two ways. First, we require data on at least 25 students for 
a 8lngle-*predlctor analysis. For a two-predictor analysis, we require 50 
students; for three predictors, 75 students, and so on. These students must 
have data on the criterion measure and all the predictors. These sample 
size requirements apply to analyses for subgroups of students as well for 
the total group. However, with as few as 25 students In a two-way table, 
there may be very few — possibly none at all — In a particular score Interval 
on the predictor. Tet, we are reporting conditional probabilities for 
students In that score Interval. We do It by using a two-way smoothing 
procedure that. In effect, borrows Information from the other score 
Intervals. And In this procedure we use a fairly strong model, only 
seven parameters to be estimated from the data. This two-way smoothing 
procedure Is our second way of dealing with small samples. 
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We deal with the problem of highly correlated predictors by emphasizing 
analyses based on single predictors and equally-weighted combinations of 
predictors. The user can request a slngle-predlctor analysis based on an 
equally-weighted composite predictor, and the system will create the 
composite predictor and perform the analysis. Also, every multiple- 
predictor analysis Includes a prediction equation based on equal weights for 
the predictors. 
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Presented at the Annual Meeting of the National Council on Measurement in 
Education, San Francisco, April, 1986 « 
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